{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial for Run Your Own Task in MT-DNN\n",
    "To run your own task with MT-DNN is actually easy with 3 steps:\n",
    "1. add your task into task define config\n",
    "2. prepare your task data with correct schema\n",
    "3. specify your task name in args for train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Add Your task into task define config\n",
    "MT-DNN adapts [yaml](https://en.wikipedia.org/wiki/YAML) file as config file format. \n",
    "Here is a piece of example task define config :\n",
    "<pre>snlisample:\n",
    "  data_format: PremiseAndOneHypothesis\n",
    "  enable_san: true\n",
    "  labels:\n",
    "  - contradiction\n",
    "  - neutral\n",
    "  - entailment\n",
    "  metric_meta:\n",
    "  - ACC\n",
    "  loss: CeCriterion\n",
    "  kd_loss: MseCriterion\n",
    "  adv_loss: SymKlCriterion\n",
    "  n_class: 3\n",
    "  task_type: Classification</pre>\n",
    "  \n",
    "We will take this \"snlisample\" task as example to show you what are these fields and add them step by step. You can find the full config file under \"tutorial\" folder as well.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add task definition for your task\n",
    "\n",
    "<pre>snlisample\n",
    "  task_type: Classification\n",
    "  n_class: 3</pre> \n",
    "\n",
    "  speicfy what task type it is in your own task, choose one from types in:\n",
    "    1. Classification\n",
    "    2. Regression\n",
    "    3. Ranking\n",
    "    4. Span\n",
    "    5. SeqenceLabeling\n",
    "    6. MaskLM\n",
    "  More details in [data_utils/task_def.py](../data_utils/task_def.py)\n",
    "  \n",
    "Also, specify how many classes in total in your task, under \"n_class\" field.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add data information for your task\n",
    "\n",
    "<pre>snlisample:\n",
    "  data_format: PremiseAndOneHypothesis\n",
    "  enable_san: true\n",
    "  labels:\n",
    "  - contradiction\n",
    "  - neutral\n",
    "  - entailment\n",
    "  n_class: 3\n",
    "  task_type: Classification\n",
    "  </pre> \n",
    "  \n",
    "  choose the correct data format based on your task, currently we support 4 types of data formats, coresponds to different tasks:\n",
    "  1. \"PremiseOnly\" : single text, i.e. premise. Data format is \"id\" \\t \"label\" \\t \"premise\" .\n",
    "  2. \"PremiseAndOneHypothesis\" : two texts, i.e. one premise and one hypothesis. Data format is \"id\" \\t \"label\" \\t \"premise\" \\t \"hypothesis\".\n",
    "  3. \"PremiseAndMultiHypothesis\" : one text as premise and multiple candidates of texts as hypothesis. Data format is \"id\" \\t \"label\" \\t \"premise\" \\t \"hypothesis_1\" \\t \"hypothesis_2\" \\t ... \\t \"hypothesis_n\".\n",
    "  4. \"Seqence\" : sequence tagging. Data format is \"id\" \\t \"label\" \\t \"premise\".\n",
    "  \n",
    "  More details in [data_utils/task_def.py](../data_utils/task_def.py)\n",
    " \n",
    "The code is using surfix to distinguish what type of set it is (\"_train\",\"_dev\" and \"_test\"). So:\n",
    "  1. make sure your train set is named as \"TASK_train\" (replace TASK with your task name)\n",
    "  2. make sure your dev set and test set ends with \"_dev\" and \"_test\".\n",
    "\n",
    "If you prefer using readable labels (text), you can specify what labels are there in your data set, under \"labels\" field."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add hyper-parameters for your task\n",
    "\n",
    "<pre>snlisample:\n",
    "  data_format: PremiseAndOneHypothesis\n",
    "  enable_san: true\n",
    "  labels:\n",
    "  - contradiction\n",
    "  - neutral\n",
    "  - entailment\n",
    "  n_class: 3\n",
    "  task_type: Classification\n",
    "  </pre>\n",
    "\n",
    "   \n",
    "  set \"true\" if you would like to use Stochastic Answer Networks([SAN](https://www.aclweb.org/anthology/P18-1157.pdf)) for your task;\n",
    "  We also support assigning different dropout prob for different task as well, please assign the prob for your task in \"dropout_p\" field. More examples please refer to other glue task def files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add metric and loss for your task\n",
    "\n",
    "<pre>snlisample:\n",
    "  data_format: PremiseAndOneHypothesis\n",
    "  enable_san: true\n",
    "  labels:\n",
    "  - contradiction\n",
    "  - neutral\n",
    "  - entailment\n",
    "  metric_meta:\n",
    "  - ACC\n",
    "  loss: CeCriterion\n",
    "  kd_loss: MseCriterion\n",
    "  adv_loss: SymKlCriterion\n",
    "  n_class: 3\n",
    "  task_type: Classification\n",
    "  </pre>\n",
    "  \n",
    "  More details about metrics,please refer to [data_utils/metrics.py](../data_utils/metrics.py);\n",
    "  \n",
    "  you can choose loss, kd_loss (knowledge distillation) and adv_loss(adversarial training) from pre-defined losses in file [data_utils/loss.py](../data_utils/loss.py), and you can implement your customized losses into this file and specify it in the task config.\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Prepare your data in correct format\n",
    "\n",
    "remember what \"data_format\" you set in config? please follow the detailed data format below, which we also mentioned above, to prepare your data:\n",
    "\n",
    "1. \"PremiseOnly\" : single text, i.e. premise. Data format is \"id\" \\t \"label\" \\t \"premise\" .\n",
    "2. \"PremiseAndOneHypothesis\" : two texts, i.e. one premise and one hypothesis. Data format is \"id\" \\t \"label\" \\t \"premise\" \\t \"hypothesis\".\n",
    "3. \"PremiseAndMultiHypothesis\" : one text as premise and multiple candidates of texts as hypothesis. Data format is \"id\" \\t \"label\" \\t \"premise\" \\t \"hypothesis_1\" \\t \"hypothesis_2\" \\t ... \\t \"hypothesis_n\".\n",
    "4. \"Seqence\" : sequence tagging. Data format is \"id\" \\t \"label\" \\t \"premise\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization and Convert to Json\n",
    "\n",
    "the training code reads tokenized data in json format. please use \"prepro_std.py\" to do tokenization and convert your data into json format.\n",
    "For example, let's give a try on example task \"snlisample\"! please try with following commands to preprocess data for this task:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>python prepro_std.py --model bert-base-uncased --root_dir tutorials/ --task_def tutorials/tutorial_task_def.yml</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example Output:\n",
    "<pre>\n",
    "07/02/2020 08:42:26 Task snlisample\n",
    "07/02/2020 08:42:26 tutorials/bert_base_uncased/snlisample_train.json\n",
    "07/02/2020 08:42:26 tutorials/bert_base_uncased/snlisample_dev.json\n",
    "07/02/2020 08:42:26 tutorials/bert_base_uncased/snlisample_test.json\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Onboard your task into training!\n",
    "\n",
    "1. Add your piece of config into overall config for all tasks\n",
    "2. if you are looking for multi-task learning, to join your new task with exsting tasks, please append your task and test_set prefix in train.py args : \"--train_datasets EXISTING_TASKS,YOUR_NEW_TASK --test_datasets EXISTING_TASK_TEST_SETS,YOUR_NEW_TASK_SETS\"; if you are looking for single task fine-tuning, please just leave your new task only in the args.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for example, we would like to finetune this \"snlisample\" task, with the sampled data in tutorials folder, here is the command:\n",
    "<pre>\n",
    "python train.py --task_def tutorials/tutorial_task_def.yml --data_dir tutorials/bert_base_uncased/ --train_datasets snlisample --test_datasets snlisample\n",
    "</pre>\n",
    "\n",
    "If you would like to add adversarial training, make sure you have \"adv_loss\" defined in task config file, and please add \"--adv_train\":\n",
    "<pre>\n",
    "python train.py --task_def tutorials/tutorial_task_def.yml --data_dir tutorials/bert_base_uncased/ --train_datasets snlisample --test_datasets snlisample --adv_train\n",
    "</pre>\n",
    "\n",
    "Example Output:\n",
    "<pre>\n",
    "07/02/2020 09:13:38 Total number of params: 109484547\n",
    "07/02/2020 09:13:38 At epoch 0\n",
    "07/02/2020 09:13:38 Task [ 0] updates[     1] train loss[1.25835] remaining[0:00:06]\n",
    "predicting 0\n",
    "07/02/2020 09:13:42 Task snlisample -- epoch 0 -- Dev ACC: 36.000\n",
    "predicting 0\n",
    "07/02/2020 09:13:42 [new test scores saved.]\n",
    "07/02/2020 09:13:44 At epoch 1\n",
    "predicting 0\n",
    "07/02/2020 09:13:47 Task snlisample -- epoch 1 -- Dev ACC: 43.000\n",
    "predicting 0\n",
    "07/02/2020 09:13:48 [new test scores saved.]\n",
    "</pre>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}